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PROGRAM EVALUATION IN ADULT EDUCATION 
Introduction 

There were several reasons behind the Wisconsin Assessment Center's 
decision to prepare this resource paper. During the past several years, there 
has been a renewed interest in the assessment of postsecoudary institutions 
and programs. Program administrators in particular have become Increasingly 
aware of the importance of evaluation. Faced with the prospect of diminishing 
financial resources, institutions need to make Informed decisions on how to 
allocate the available funds. Consequently, for many programs, evaluation has 
become more of a necessity for survival than a luxury. 

While most program administrators and staff have had some courses in tests 
and measurement and some experience with educational research, few have had 
the training to conduct a systematic evaluation of their program. The 
Wisconsin Assessment Center's response to this problem has been to develop a 
resource guide which would focus on the process of program evaluation, 
particularly as it relates to programs which specialize in meeting the needs 
of the nontradltlonal aged student. 

This paper provides a review of a wide range of books, guides, journal 
articles and other professional literature associated with program evaluation 
in adult and higher education. It focuses on some of the basic questions 
associated with program evaluation. For example, what purposes does program 
evaluation serve? What are the major program evaluation models used? In what 
ways, if any, is program evaluation in adult education different from the 
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evaluation of other types of programs? What are the essential elements of a 
program evaluation? How can the evaluator Improve the chances that the 
results of the evaluation will be used? 

In addition, a concerted effort has been made to develop a document which 
Is directed at the adult education administrator, particularly the novice 
program evaluator and practicing educator who might be Interested In the 
process of program evaluation. However, this paper Is not Intended to be a 
planning guide or textbook on how to do a program evaluation. It Is designed 
to provide an Introduction and serve as a resource for Individuals faced with 
the problem of where to begin their study of the concepts and techniques of 
this swlfMy changing field. 

The question of what documents to review, what theories to discuss and 
what models to emphasize was not an easy one to address. Even a brief foray 
into the program evaluation literature reveals an immense amount of 
information including hundreds of books and thousands of journals articles • 
Out of necessity, the criteria used to select the books, guides, workbooks and 
journal articles which appear in this paper was somewhat arbitrary. In 
general the journal articles selected were those abstracted in the Cumulative 
Index for Journals in Education between 1981 and 1985, CUE is perhaps the 
most comprehensive and appropriate source of program evaluation literature. 
Books were selected to fairly represent what is currently available in the 
field. Most have been published in the last ten years but no attempt was made 
to do an exhaustive search and many important works may have been overlooked. 
The guides and workbooks selected for review include a wide selection of 
how-to-do manuals. They range from the fairly simple and straight forward to 
the lengthy and complex. All of them, to some degree, can be useful to the 
adult program evaluator. 
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The paper has been divided Into four major sections. The first section 
provides an overview of the major purposes for doing a program evaluation in 
adult education. Section two includes a brief description of four selected 
evaluation models which appear to be particularly applicable to adult 
education programs. The third section provides a critique of several 
commercially published "hoi-^to-do-an-evaluation** packages and guides which can 
be applied to postsecondary adult education. The final section consists of an 
analytical summary of books and guides pertaining to program evaluation and an 
annotated bibliography of selected journal articles and books which roughly 
covers the professional literature for the past five years. 

Section I, Purposes for Program Evaluation 
The prerequisite of any program evaluation should be delineation of a 
clear rationale for why an evaluation is be-^ng conducted. For example, a 
program director may be called upon by a governing board or administrator to 
evaluate his/her program and make recommendations about new policies and 
directions. Another director may have difficulty securing future funding 
support for a program without the availability of relevant program evaluation 
data. 

One way to conceptualize these and other purposes is to divide them into 
basically two categories. The first category includes reasons associated with 
"program improvement." The second involves the determination of the 
"effectiveness" of a program, Scriven (1967) operationalized this distinction 
by differentiating a summative evaluation from a formative one. In summa^ive 
evaluation, intormation is used to prove something, satisfy accountability or 
make a judgement about the overall quality of a program. Formative evaluation 
is used to modify and improve a program and is frequently used to provide 
feedback to staff and administrators while the program is in operation, 
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Uslng this distinction as the basis for a more detailed description of 
purposes, Deshler (1984) divides the purposes for program evaluation into four 
general areas • In general, the first two areas can be considered forms of 
summative evaluation while the third and fourth provide specific examples of 
formative evaluation purposes • However, in actual practice formative and 
summative evaluation frequently overlap^ 

Using this framework, the four major reasons for doing a program 
evaluation are (1) to satisfy externally mandated accountability requirements, 
(2) reduce uncertainty, (3) improve practice and (4) contribute to 
social/political influences. More specific reasons for doing an evaluation 
include the following: 

1, Accountability - providing requested evidence of performance; justify 
or advocate a position that supports program continuation, expansion 
or changes. (Formative Evaluation) 

2, Reduce Uncertainty - increase credibility and support for unpopular 
decisions; add or drop specific strategies or techniques; continue or 
discontinue a program; allocate resources among competing programs; 
initiate a similar program somewhere else. (Formative Evaluation) 

3. I mprove Practice - increase efficiency and reduce waste; identify and 
detect malfunctions and defects in procedures and program designs; 
develop new approaches and alternative procedures; adjust programs to 
changing situations, clients and/or resources; understand a failure 
to prevent future failures. (Summative Evaluation) 

4. Social/Political Influences - stimulate informed political dialogue 
and challenga simplistic views; resolve political conflicts; address 
issues and add to social science knowledge pool; fill in conceptual 
gaps. (Summative Evaluation) 
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Section II. Selected Program Evaluation Models 
There are a variety of models one can use in conducting a program 
evaluation* To a great extent, the model chosen is influenced by one's 
philosophy abour. evaluation — although other factors including time, 
resources, evaluation expertise and availability of staff strongly influence 
the procedures used* Most program evaluation experts agree that there is no 
one best available model* As a result, it is necessary for the program 
evaluatcr to select a model which matches the requirements of a particular 
situation to produce evaluation findings which are most likely to accurately 
appraise a program's merits and shortcomings and to provide information about 
its goals, activities, outcomes, impact and costs* 

Specific approaches to program evaluation have been summarized and 
compared in several sources* House (1978), Madaus, Scriven and Stuff lebeam 
(1983) and Worthen and Sanders (1973) are particularly good references for the 
reader who is interested in gaining a bioad perspective on the evaluation 
models which can be used within the context of adult education* Considering 
the wide variety of program evaluation models discussed in the literature, I 
have selected four which I consider to be particularly interesting or 
appropriate within the context of adult education* Many of the evaluation 
guides and other commercially published resources described in the third 
section of this paper are based in whole or in part on the models briefly 
described below. These models include the CIPP Evaluation Model, Discrepancy 
Evaluation, Responsive Evaluation and the Judicial Evaluation Model* 
The CIPP Evaluation Model 

The CIPP Evaluation model (Madaus, et al., 1983; Stufflebeam, 1969, 1971; 
Stake, 1967) was conceptualized by Daniel Stufflebeam as a result of attempts 
to evaluate projects that had been funded through the Elementary and Secondary 
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Educatlon Act of 1965. CIPP Is essentially qualitative In nature and 
complements the emphasis many programs place on goals and objectives. 
Elements of this approach were first popularized by Tyler (1969) and adopted 
by most federal and state funding agencies. There are four basic components 
In the model: context, Input, process and product evaluation. Context 
evaluation Is designed to focus on how the program "fits** Into the larger 
system or organization and the methods used to meet the program's goals. The 
results of the context evaluation generally lead to decisions about whether to 
Introduce some kind of a change in the program. 

Input evaluation measures the aaount of effort expended to meet the 
program's goals. Such an evaluation would query personnel about the amount of 
time spent in various activities — the program director might be asked to 
itemize costs and other institutions might be studied to determine how they 
have implemented similar programs. Input evaluations are used primarily to 
decide whether the amount of time and resources Invested in a program are 
compatible with the anticipated outcomes. 

Process evaluation focuses on the design of the program, its 
implementation and the recording of procedural activities. The general 
purpose of process evaluation is to assist in the proper implementation of the 
program, identify defects and refinement of the program design and 
procedures. In some Instances, staff and program administrators keep a log of 
the actual process of implementing the program for later use in interpreting 
outcomes. 

Product evaluation concentrates on the collection of data, descriptions 
and judgements concerning outcomes and relates them to the program's 
objectives. Findings from context, input and process evaluations can also be 
Incorporated into this evaluation component. Product evaluation is 
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particularly useful when It Is necessary to gather information which will be 
used to make a decision to continue or modify a program^ 

An important factor when considering the use of the CIPP model is that a 
specific evaluation can include just one or a combination of the four 
components. The purpose or rationale for doing the evaluation ultimately 
determines which elf;ments are used. Since its inception in the mid-1960s CIPP 
has become one of the most widely applied evaluation models used in 
educational settings* 
The Discrepancy Evaluation Model 

The Discrepancy Evaluation model, developed by Andres Stelnmetz, (Madaus, 
et al*, 1983; Stelnmetz, 1976, Tyler, 1961), is based on the premise that in 
all types of evaluation we inevitably make comparisons* Use of this mode} 
would be particularly appropriate for programs which are able to establish 

standards" which can be fairly easily quantified. The model has three basic 
evaluation phases. The first phase Involves the articulation of the program* s 
standards, that is generating a list or descriptlor. of the qualities the 
program should possess. These standards can be derived from a historical 
review of documents related to the program* s developmfmt or from interviews 
with past and present administrators and staff. 

The second phase Is concerned with the actual evaluation of the 
characteristics of the program. This phase is similar to the process 
component of the CIPP model and Stelnmetz has borrowed heavily from 
Stufflebeam and his associates to develop tftis aspect of his own model. 

The third phase involves a comparison between the standards articulated in 
the first element and the performance measured in the second. The discrepancy 
between these two elements provides the basis for making a judgement about the 
overall effectiveness or worth of the program being evaluated. 
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The Discrepancy Evaluation Model offers a straightforward, pragiuuLic 
approach to a wide variety of evaluation needs. It can be utilized by ^-^nlt 
education teachers to evaluate their dally classroom activities or by the 
director of a large and complex adult education program. The Discrepancy 
Evaluation Model Is particularly useful when program administrators wish to 
place an emphasis on self-evaluation and systematic program Improvement. 
Responsive Evaluation 

Robert Stake (1973) contends that the focal point of program evaluation 
should be on the Identification of the ''real Issues" which prompted the call 
for an evaluation of the program In question. Therefore, the clients being 
served by the program, program sponsors and program staff are all Interviewed 
or surveyed to determine which questions or Issues should be addressed during 
the evaluation process. The major difference between this model and most 
others is that the formulation of evaluation questions does not begin with an 
identification of the program's stated goals or activities. Rather, it is the 
perceptions of those individuals, who have a vested interest in the program, 
which determine what questions will be formulated and how the evaluation will 
proceed. 

The basic assumption underlying this approach to program evaluation is 
that evaluation will be effective and useful only if it is responsive to the 
Interests and needs of the Individuals most affected by the program's 
existence. As a result, the evaluation plan is designed to be flexible enough 
to allow for changes at any time during the evaluation process. Frequently, 
large expenditures of resources are allocated to observing the program. 
Qualitative and naturalistic evaluation techniques are emphasized as a way of 
shoring up, reinforcing and humanizing the evaluation process. Responsive 
evaluation is particularly useful for formative evaluation or when the primary 
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purpose Is to understand. In detail, the program's activities, strengths and 
shortcomings. Madaus et al., (1983) provide an excellent summary of this 
evaluation approach. 
J udicial Evaluation Model . 

The Judicial Evaluation Model (Cuba, 1978; Madaus, 1983; Wolf, 197A, 1978; 
Wolf & Tymltz, 1977) was conceptualized In the early 1970s, by Robert Wolf, as 
a method which would be useful for administrators who need to reach some 
decision regarding an educational program and want to go beyond the gathering 
of objective data. The model allows for the gathering of both objective and 
subjective Information and Is patterned after the legal model which places a 
premium on human testimony and judgement. It establishes a systematic 
procedure for Inquiry Including criteria for classifying, evaluating and 
presenting evidence in a clear, concise and reasonable manner. 

Two investigative teams are formed to evaluate the program in question. 
One team is charged with building a case for a reduction or elimination of the 
program while the other is asked to gather evidence in support of its 
continued existence. Each team Includes a case ana"*yst who serves as the team 
supervisor and a case presenter who has the responsibility of actually 
presenting the case in "court." A forum moderator enforces the rules 
established for the proceeding and rules on objections. Finally, a 
clarification panel is selected to consider the evidence and present a written 
statement of their recommendations. 

The judicial evaluation model is implemented in four stages. Stage one is 
an exploratory stage designed to identify as broad a range of Issues as 
possible. The naturalistic inquiry paradigm is used heavily in this stage of 
the process. During the second stage, the Issues identified are placed in 
order of priority and pooled to reduce them to a manageable size. Again, 
decisions are made through the extensive use of the strategies of naturalistic 
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Inqulry. The third stage involves building cases and preparing arguments for 
case presentation. As in judicial court proceedings, both teams share their 
information and respective plans of action. In the final stage there is a 
public presentation of the data and other information collected. Case 
presenters make their cases and call witnesses. Direct and cross examination 
of witnesses are engaged in and opening and closing arguments are presented. 
Based on the evidence presented, the panel makes its decision and 
recommendations regarding the program being evaluated. 

Ine Judicial Evaluation model can be particularly useful for dealing with 
policy-level problems. However, implementing this model in its entirety is a 
complex and complicated task. This model would probably be most useful for 
adult educators who want to be sure to incorporate both the positive and 
negative aspects of a particular program into their evaluation plan along with 
a heavy emphasis on naturalistic inquiry methods and human judgement. 
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Section III. Comasrcially Published Guides/Evaluation Packages 
Since the late 19608, the emphasis on program evaluation in educational 
practice has greatly increased. The decide of the 1980s has brought added 
importance to academic program evaluation* The projected decline in 
traditional-aged students and its impact on university enrollment will 
continue to be a significant problem* In addition, the demand for 
accountability and fiscal austerity is expected to continue* Institutional 
administrators are confronted, now more than ever, with problems of 
maintaining the quality of academic programs. They are also being asked to 
provide support for new programs to meet the changing needs of students and 
society* 

While program administrators are rarely expected to be evaluation experts 
they are expected to be able to determine a good evaluation plan from a poor 
one* Limited time and resources may rule out calling in some outside experts 
to do a comprehensive program review. Consequently, a great deal of time and 
energy can be saved by consulting materials which are designed to assist an 
administrator or evaluation committee in the organization and implementation 
of an evaluation plan* 

Many books have been published over the past few years on the evaluation 
of educational programs* The majority of these, however, are written in 
highly technical language and are much too cumbersome to be of benefit to 
individuals who are concerned with how to do a program evaluation, but do not 
have expertise in the area* The general purpose of this section of the paper 
will be to provide a description and critique of several program evaluation 
guides and other materials which are currently available* These "packages" 
are all designed to help guide the program administrator or evaluator through 
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the program evaluation process. Most of these materials were prepared by 
nationally recognized research and evaluation centers with extramural support 
from state and federal agencies. Many Include a description of the program 
evaluation process and Include worksheets, "do-it-yourself** exercises and 
other aids so that the user can effectively conduct a program evaluation. All 
ot the materials described can be adopted for use by adult education programs. 

Each critique Includes four major sections: (1) a general description of 
the materials, (2) Intended use, (3) general comments and, (A) Information on 
how to obtain the materials. 
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Postsecondary Institutions and the Adult Learner: 
A Self-study Assessment and Planning Gulde > 
by Commission on Higher Education and the Adult Learner 
General Description 

The Assessment Guide Is a two-part self-study Instrument which Is designed 
to help ati Institution or program administrator organize an evaluation of a 
program which serves adult learners. Part I of the guide provides a brief 
Introduction and an annotated bibliography of references which may be helpful 
in academic planning in the field of adult education. 

Part II of the Guide contains the Self-study Instrument. There are ten 
modules which have been conceptualized as a comprehensive review of adult 
learner programs, policies and practices. Each section of the guide contains 
descriptors which typify good policy or practice at institutions where adult 
learners are well served. The format of the guide Includes checklists and 
rating scales to assist working groups in the evaluation of the selected 
program. 
Intended Use 

The Guide "is designed for use by college and university presidents with 
the cooperation of key administrators to ascertain the current effectiveness 
of their institution, or a unit within their institution, in serving adult 
learners. It may also be used to assess institutional readiness to serve an 
adult clientele. Further, it can be an aid to institutional self--study for 
purposes of accreditation or state approval. This guide is intended to be 
used as part of the assessment and planning process to provide basic 
information for institutional planning with respect to instruction and 
services for adult learners. 
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Comments 

This guide was developed as part of a larger project to develop methods 
which could be used by postsecondary institutions to evaluate programs which 
are specifically targeted to the adult learner. Several institutions across 
the country have used this guide under the direction of the organizations 
which developed these materials. Preliminary indications are that 
institutions have found the information provided to be very useful. The 
instrument itself provides excellent definitions and planning notes which can 
be placed within the context of a more general program evaluation plan. The 
Guide is probably one of the better tools currently available for customizing 
an evaluation plan to meet the needs of a program serving the adult learner. 
How to Obtain Materials 

The self-study assessment and planning guide is published by the 
Commission on Higher Education and the Adult Learner, 10598 Marble Farm Court, 
Columbia, Maryland 21044. The cost in February 1985 was ^17. 25. 
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Program Evaluation Kit , 
by Carol Taylor Fltz-Glbbon and Lynn Lyons Morris 
General Description 

The Program Evaluation Kit was developed at the Center for the Study of 
Evaluation, University of California, Los Angeles* The entire project was 
funded by the National Institute of Education, United States Department of 
Health, Education and Welfare, The Kit Includes a series of eight practical 
guides covering various aspects of program evaluation* The first book In the 
series. The Evaluator' s Handbook, Is meant to orient the reader to program 
evaluation. The remaining volumes address seven major evaluation problems: 

a. How to Deal With Goals and Objectives 

b. How to Design a Program Evaluation 
How to Mea3ure Program Implementation 

d. How to Measure Achievement 

e. How to Measure Attitudes 

f. How to Calculate Statistics 

g. How to Present an Evaluation Report 

Each guidebook provides Instructions, checklists, worksheets and examples 
of the topics discussed. 
Intended Use 

"The Program Evaluation Kit Is a set of books Intended to assist people 
who are conducting evaluations of educational programs. The scope of Its 
potential use Is broad. Because It comprises a set of step-by-step procedural 
guides, the Kit can advise a person conducting elaborate evaluations of 
far-reaching and many-faceted programs or It can help people as they gather, 
analyze, and Interpret Information for almost any purpose-whether it be 
surveying peoples* attitudes, observing a program In action, or measuring 
student achievement** (The Evaluator's Handbook, p, 5), 
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In addition to suggesting step-by-step procedures, the Kit Introduces and 
explains concepts and vocabulary common to evaluation. You can therefore use 
It for training or staff development* The Kit Is intended to be useful to 
people with extensive experience In evaluation as well as those encountering 
program evaluation for the first time* 
Comment? 

This Is an excellent series of guides which can be used effectively by 
Individuals with a limited background in program evaluation* However, persons 
without at least an undergraduate statistics course and a solid background In 
tests and measurements may find It difficult to use the material In some of 
the books. In general, however, I would highly recommend this series to the 
adult educator who Is confronted with a wide range of evaluation problems and 
Issues. With only a moderate Investment of time the worksheets and narrative 
Instructions provide an excellent framework for conducting a high quality 
evaluation of an educational program. 
How to Obtain Materials 

The Program Evaluation Kit Is published by Sage Publications, Inc., 275 
South Beverly Drive, Beverly Hills, California 90212. The current price for 
the complete kit Is ^59. 95. 
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An Evaluation Primer and An Evaluation Primer Workbook: 



Practical Exercises for Educators. 



by Arlene Fink and Jacqueline Kosecoff 



General Description 

The Primer has been organized into eight chapters. The first chapter 
includes introductory information which the authors consider essential in 
helping the evaluator move from thinking about program evaluation to doing 
program evaluation. The remaining chapters discuss methods and procedures for 
formulating credible evaluation questions (chapter 2), constructing evaluation 
designs (chapter 3), planning information collection (chapter 4), collecting 
evaluation information (chapter 5), planning and conducting information 
analysis activi* les (chapter 6), reporting evaluation information (chapter 7) 
and managing an evaluation (chapter 8), The companion Workbook provides a 
series of exercises which are Intended to provide practice in applying 
evaluation skills to education programs. Each chapter in the Workbook 
corresponds to a chapter in the E valuation Primer , 
Intended Use 

The major purpose of these two volumes is to enable the reader to conduct 
and interpret evaluations of social programs. The Primer and Workbook are 
based on the premise that "the evaluator' s main task is to provide information 
that is technically sound and not to make decisions about how to use it" (p. 
ix). The primary Intended users are professional educators, school 
administrators and evaluation specialists who are charged with conducting a 
program evaluation, particularly the evaluation of social programs. 
Comments 

These materials provide extremely well organized and concise step-by-step 
procedures for conducting a program evaluation. The authors place a great 
deal of emphasis on objective evaluation procedures including the use of 
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quasl-experimental designs and statistical analysis of the evaluation data. 
Although these materials were designed for evaluating social programs, the 
adult education administrator or program director would luJfe little difficulty 
adopting the materials for their use. All things considered, the Primer and 
Workbook could be very useful for the novice program evt^luator. 
How to Obtain Materials 

The Evalua^.ion Primer and Workbook is published by Sage Publications, 
Inc., 275 South Beverly Drive, Beverly Hills, California 90212. The cost in 
1985 was ^9.95 for softcover book. 
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Program E valuator's Guide and Workbook on Program Evaluation 



by Alexander I. Law and Carolyn M. Fowle 



General Description 

This guide provides a detailed description of the steps Involved in 
planning and carrying out a program evaluation. The guide was developed as 
part of the California Evaluation Improvement Project under the auspices of 
the California State Department of Education. The Guide contains eight major 
sections: determining the purposes and requirements of the evaluation, 
development of a program evaluation plan, determination of evaluation and 
sampling, the selection and development of evaluation instruments, data 
collection, analysis of evaluation data, reporting of evaluation results and 
application of evaluation findings. 

Each section Includes numerous examples, charts and checklists to assist 
the program evaluator throughout the entire evaluation process. The inclusion 
of brief case studies, exercises, random numbers tables and excerpts from 
actual evaluation reports are also Included to clarify the content. 

The Workbook is Intended to accompany the Guide and contains review 
sheets, practical examples of the major cojiponents of an evi?luatlon plan, 
worksheets and data analysis forms. Each section of the Workbook corresponds 
to a chapter in the Guide and can serve as a handy reference and review of the 
concepts covered by these materials. 
Intended Use 

The Guide have been used in conjunction with two and five-day workshops to 
improve program evaluation practices among educators. The Guide was prepared 
to give the practitioner an Increased appreciation of program evaluation 
practices and higher-level skills in the key evaluation functions. The Guide 
is not particularly technical in nature, consequently, it is particularly 
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approprlate for workshops or classes which are comprised of educators and 
adult education practitioners rather than evaluation or measurement 
specialists. 

The Workbook has two purposes. It can be used as a learning and 
Instructional aid while one masters the procedures, techniques and methods of 
program evaluatlon« It can also be used to help an evaluation team keep 
complete records of the Important Information related to the program 
evaluation* 
Comment 

This is another excellent set of materials which describes in great detail 
the major components of the evaluation process. The basic philosophy behind 
the materials seems to be that the practitioner is in the best position to 
design and Implement an effective program evaluation. Despite its emphasis on 
the evaluation of public school programs, this guide and workbook can be used 
very effectively to assist the adult education practitioner to properly 
collect information that will indicate how well a program is meeting its 
objectives, how well the program is being Implemented and other information 
required to make Informed decisions about the program being evaluated. 
H ow to Obtain Materials 

These materials were originally published by the California State 
Department of Education but is distributed by the Educational Testing Service 
Publications Order Service, CN 6736, Princeton, N.J. 08541-6736. The cost of 
the Guide is ilZ.OO, The workbook is ^8.00. 
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Program Evaluation: A Practitioner^ s Guide for Trainers 
and Educators^ Sourcebook/Casebook and Design Manual 
by R.O. Brlnkarhoff , D.M. Brethower, T. Hluchy, and J.R. Nowakowskl 



The Practitioner* s Guide was developed by the Evaluation Training 
Consortium (ETC) project at the Evaluation Center, Western Michigan 
University. The ETC project was funded by the U.S. Office of Special 
Education from 1972 to 1982. 

The package consists of two volumes. The first volume is a sourcebook, 
which contains examples, guidelines, criteria and checklists which can be used 
to Implement each of seven key evaluation functions: 



(1) focusing an evaluation and clarifying its purpose 

(2) designing an evaluation 

(3) collecting Information 

(4) analyzing iiiformation 

(5) reporting: Interpreting and using evaluation findings 

(6) managing evaluation activities 

(7) evaluating evaluation efforts 



The Casebook (which is in the same hardcover volume as the Sourcebook) is 
a collection of twelve case studies about evaluation applied to real-life 
programs in different settings. These case stucUes are Included to show how 
each of the seven evaluation functions can be Implemented in different 
settings. 

The soft-covered Design Manual contains a programmed set of worksheets, 
directions, and guidelines for designing a program evaluation. The manual 
assists the user in the production of a document which covers each facet of 
the overall evaluation design. 



General Description 
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Intended Use 

According to the authors, this set of materials Is Intended for use In 
training, teacher education and other professional programs and projects In 
agencies and postsecondary Institutions, They were designed to be used by 
Individuals or groups for their own program evaluation use and can also be 
used to train others. 
Comments 

This guide Is obviously the result of a great deal of time and effort* 
The authors present a fairly advanced treatment of how to conduct a program 
evaluation and have designed the materials around a commonly accepted division 
of the key evaluation functions. The comprehensiveness of the materials 
provides the potential user with a graat deal of Information, However, I 
suspect that only the experienced e valuator could make complete use of the 
Information provided* An Individual looking for a concise, easy to understand 
discussion of the evaluation process will probably not find this 
practitioner's guide useful. The evaluation consultant or administrator with 
some knowledge and experience with the program evaluation process will 
probably find the guide to be very Informative and a fine addition to their 
reference library* 
How to Obtain Materials 

These materials can be obtained by contacting Kluwer-Nl jhof f Publishing, 
190 Old Derby Street, Hlngham, Massachusetts 020A3* In 1985 the Sourcebook 
and Cacebook was $35* 95 and the Design Manual was $17* 95* 
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Section IV. Selected Program Evaluation Literature 
ThlB section provides a summary of reference Information on the general 
topic of program evaluation In higher education using two formats: an 
analytical simmiary chart aad an annotated bibliography. The documents 
summarized In the chart Include only books and guides pertaining to program 
evaluation. All of them are relevant to evaluation In adult education, 
although not all of the publications can be considered pare of the adult 
education literature. Summarizing the Information In this manner''" Is an 
attempt to provide the reader with an index to the type of program evaluation 
Inforaatlon found In each publication. The chart is divided into several 
categories which correspond to key elements in the evaluation process. 
Publications are listed alphabetically by author. An "x** on the chart 
indicates that the publication contains a fairly detailed description of the 
particular aspect of program evaluation indicated by the column headings. A 
complete bibliographic citation can be found in the final section of this 
document. 

The annotated bibliography was drawn from the Current Index of Journals in 
Education (CIJE) database of the Educational Resources Information Center 
(ERIC). The bibliography Includes journal articles which cover several 
aspects of program evaluation with an emphasis on evaluation of adult higher 
education programs. The listing is not intended to be exhaustive and, in 
fact, only covers the years 1981-1985. However, the sources listed appear to 
representative of the available literature 



The format for this summary is similar to one used by Sork (1984). 
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Flnally, the reader should keep in mind that the number of professional 
publications on the topic of program evaluation have expanded tremendously 
since the early 1970s. In addition to a wide variety of books and manuals, 
several periodicals which publish articles pertaining to program evaluation 
are now available. Some of the better publications include: Evaluation News , 
The Journa l of Evaluation and Program Planning , New Directions for Program 
Evaluation , Research in Higher Education , New Directions for Institutional 
Research, and Evaluation and the Health Professions . Some organizations also 
sponsor bulletins or newsletters to update people on the current happenings in 
the field of program evaluation. One such bulletin is the Evaluation News 
Quarterly Bulletin . How to Evaluate Education Programs is a newsletter 
published by Capital Publications in Washington, D. C, which provides concise 
summaries of particular aspects of program evaluation. 
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ANALYTICAL SUMMARY OF PROGRAM EVALUATION LITERATURE 
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H 




X 


Rutman, 1984 


SP 




X 
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ANALYTICAL SUMMARY, CONT'D 
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X 
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X 


X 
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X 
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X 
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X 
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X 
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SP 


X 
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I 

I 



KEY TO ABBREVIATIONS 



General Context Abbreviations 



Data Collection Procedures Discussed 
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ACE = Adult Continuing Education 
HE « Higher Education 
POST « Elementary/Secondary Education 
GEN « General, All Education Levels 
SP » Evaluation of Special Programs 



A » Written questionnaire 

B « Interviews/discussions 

C « Report/document analysis 

D « Observation 

E * Tests 

F » Self-report 

G » Supervisor's rating 

Cost-benefit analysis 
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Annotated Bibliography of Selected References 

Assessing Student Outcome as Part of the School Self-study. (1985) North 
Central Association Quarterly ; 60(1), 14-20* 

Committee oc schools entitled, **Assessing Student Outcomes a Guide to 
Incorporating with School Self-study Activities and the Teen Visit and 
Evaluation of the Extent to Which Student Outcome Objectives are Being 
Met." Presents procedures for assessing direct or school/student and 
instructional program outcomes. 

Backer, Thomas E. et al. (1980) Information Resources for Program 

Evaluator* s. Evaluation and Program Planning; an International Journal ; 
3(1), 25-33, 

Included are listings of important books, journals and newsletters* 
Brief descriptions of the major professional societies and the program 
evaluation tield (including membership information) a listing of resources 
for evaluation training materials and a brief discussion of targeted 
research on evaluation. 

Baker, Octave V. (1979) Criteria for the Evaluation of Career Education 

Activities. Journal of Research and Development in Education ; 12(3), 9-13. 

This article presents the criteria developed by the American Institute 
for Research to select, evaluate exemplary career education projects across 
the nation for submission to the Joint Dissemination and Review Panel of 
the Office of Education and the National Institute of Education, 

Ball, S. (1982). The perils of program evaluation. Contemporary Education 
Review , 1(3), 166-73. 

In reacting to five recent books on program evaluation, the author 
discusses eight evaluation perils: poor preparation of evaluations, 
programs with unrealistic goals, a mix up of models, interpersonal 
conflicts, the morass of measurement, answering evaluation questions, the 
secondary evaluator as sniper and hostility for program staff. 

Ball, Samuel, (1981) Outcomes the Size of the Impacts and Program 
Evaluation. New Directions for Program Evaluation ; 9, 71-86. 

The variety of context, the politics of each situation and the 
availability of resources all dictate variations and assessing outcomes. 
Seven major principles of program evaluation are considered along with a 
questioning note to the ambitious who expect large program effects, 

Barak, Robert J, (1977) Program Reviews by State-wide Higher Education 
Agencies. Ne^^ Directions for Institutional Research ; 16 (Increasing the 
Proaccountability of Higher Education); 4(4), 67-90. 

The rationale for state level program reviews in higher education is 
illustrated by case studies in Florida and New York. Different approacl^es 
to evaluation, reviews of existing programs and agencies structure and 
responsibilities are also discussed. 
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Barnatte, J. Jackson. & Thompson, John C, III. (1979) A Descriptive 
Assessment of the Effect of Evaluations on Instruction. Studies in 
Educational Evaluation ; 5(1), 77-86. 

A suirvey examines secondary school teacher's perceptions of the affects 
of program, student and teacher evaluations on instXMctional improvement. 
Results indicated that student evaluation was most effective in influencing 
teachers to revise their instruction* Teacher and program evaluations 
appear to be ineffective as changed agents. 

Bogue, E. Grady. (1980) State Agency Approaches to Academic Program 
Evaluation. New Directions for Institutional Research , 27 (Academic 
Program Evaluation); 7(3), 69-87. 

With artistic leadership state agency roles can complement those of 
institutions and their governing boards. Enhancing the renewal function of 
program evaluation and insuring that evaluation centers on the improvement 
of educational effectiveness. 

Boruch, Robert F. (1979) Educational Program Evaluations: Some Implications 
for Evaluation Policy, a Summary. American Behavioral Scientist ; No. 4, 
23(2), 275-96. 

Presents an overview of the development of evaluation and comparative 
testing techniques. Topics discussed include case studies of evaluation 
and other cultures, objectives of the evaluation, problems, uses, 
implications, success rates and design principles 

Brinkerhoff, R. 0. (1981). Making evaluation more useful. Training and 
Development Journal , 35(12), 66-70. 

This article summarizes the major concepts and approaches of an 
evaluation training consortium project, showing how they relate to 
evaluation in a human resources context. 

Burstein, Leigh. (1984) The Use of Existing Data-Bases and Program 
Evaluation and School Improvement. Educational Evaluation and Policy 
Analysis ; 7(2), 307-18. 

The purposes are opposed to comment on practice in the use of existing 
data-bases and program evaluation and school improvement and to explore 
directions for increased and improved use. 

Caulley, D. N. (1983). Document analysis in program evaluation. Evaluation 
and Program Planning; An International Journal , 6(1), 19-29. 

An introduction and overview of document analysis and retrospective 
data collection techniques are presented. 

Cietz, James E. (1981) Program Reassessment, Reduction and Redirection. New 
Directions for Community Colleges ; 9(1), 59-67. 

Discusses the importance of occupational program reassessment and 
social reallocation program planning. Provides suggestions regarding 
reassessment planning, goals and systematic implementation. Recommendations 
of a reassessment structure that promotes objectivity, identifies the areas 
that should be evaluated and provides examples of measures of productivity 
and program resources and activities. 
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Clowes, A. (1981) • Community college program review: A model. Community 
College Review , 9(2), 3-9, 

The author proposes a community college review model which promotes 
institutional autonomy and satisfies accountability demands. Recommends 
that community colleges use a needs/access model rather than the 
quality/excellence model appropriate for universities. Proposes an open 
and flexible review process, epecific products and seven evaluation 
criteria for all re^ew programs. 

Czebisesi, Michael* (198^) Program Evaluation: A Qualitative Planning 
Tool. Planning and Changing ; 15(3), 144-51. 

The program evaluation process at the University of Houston involving 
self-studies by academic departments and review by outside evaluation teams 
is described. The process considers educational goals and objectives as 
well as the quality of students, faculty and facilities. 

Delves, A.R. & Watts, John. (1979) A Year of Evaluation. Forum for the 
Discussion of New Trends in Education ; 22(1), 27-29 » 25. 

This article notes the trendts in Australia towards school base decision 
making through school/community counsels and describes the school 
evaluation conducted by the Counsel of the Huntingdale Technical School in 
Milboume. 

Deshler, David. (1984) An Alternative Approach to Evaluation and Continuing 
Education. New Directions of Continuing Education ; 23, 3-14. 

A brief history of educational evaluation efforts and trends with 
definitions and distinctions between summative and formative evaluation 
provides an overview of alternative methods and a rationale for selecting 
formative evaluation when appropriate. 

Diamond, Esther E. (1985) Development of the Joint Committee's Standards for 
Evaluations of Educational Programs, Projects and Materials. Measurement 
and Evaluation in Counseling and Development ; 18(2), 51-57. 

Discusses the need for standards in the evaluation of educational 
programs, projects and materials and describes the development of such 
standards by a joint committee representing 12 organizations with widely 
d.' "ferin? constituencies. 

Dillon, J. T. & Starkman, S. S. (1981). A -Model" Approach to Evaluation of 
Teacher Education Programs. Education , 101(4), 3^6-371. 

This teacher education program evaluation model incorporates six 
features: program modification, field based perspective, longitudinal 
approach, involvement of related program personnel, reliable and 
comprehensive data and examination of importance in values as well as 
effectiveness. Operation of the model and reaction of the participants and 
implication of practices are described. 
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Elchelbergerr, Tony. (1979) Evaluating Educational Program: Some Basic 
Considerations and Criteria. CEDR-Quarterly ; 12(1), 16-19. 

Four fundamental Issues and program evaluations are discussed; (1) 
evaluations Involve stake holders with adverse Interests; (2) data Involve 
some degree of subjectivity; (3) meaningful Interpretation of results 
requires understanding of the assumptions on which results are based; and 
(4) programs are dynamic and require flexlblle evaluation. 

Fortune, J. C. & Forbes, R. (1982). Utilization of evaluative Information. 
Education ; 103(2), 179-182. 

The authors discuss the utilization of evaluative information In terms 
of the dimensions of evaluation, pitfalls of e'Jaluatlon^ and facts 
concerning useful evaluation. 

Friedman, Myles I. & Anderson, Lorln W. (1979) Evaluating to Solve 
Educational Problems: an Alternative Model. Educational Technology ; 
19(9), 16-23. 

A 19 step general evaluation model Is described through lt*s four 
stages: Identifying problems, prescribing program solutions, evaluating 
the operation of the program, and evaluating the effectiveness of the 
model. The role of the evaluator and discussion making Is also explored. 

Good, H.M. et al. (1980) Modeling and Evaluation. Studies In Higher 
Education; 5(1), 33-43. 

The evaluation of educational programs In higher education needs a 
better theoretical base than It has had. Proposals as to what that base 
that should encompass are discussed and an approach that Involves a more 
adequate conceptual and procedural model is explored. Focus Is on a 
system* s view point. 

Gordon, Sheila C. & Helnemann, Harry N. (1980) Structuring an Internal 
Evaluation Process. Journal of Cooperative Education ; 16(3), 47-54. 

The design of an Internal program evaluation system requires: (1) 
formulation of program, operational and Institutional objectives; (2) 
establishment of evaluation criteria; (3) choice uf data collection and 
evaluation techniques; (4) analysis of results, and (5) Integration of the 
system Into the mainstream of operations. 

Halasz, I. M. (1982). Evaluating vocational education programs In 

correctional Institutions. Journal of Correctional Education , 33(4), 7-10. 

Discusses the eight steps to be used for evaluating vocational 
education prt)grams In correctional Institutions: (1) determine purposes, 
(2) Identify evaluation audience, (3) determine questions that need to be 
answered, (4) select methods, (5) Identify staff, (6) collect data, (7) 
analyze data, and (8) reporting results. 
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Hanson, Alan (1981) Use of ^Standards of Practice** In the Design and 
Evaluation of the Continuing Education Program. American Journal of 
Pharmaceutical Education ; ^4(1), 56-60. 

To demonstrate the potential use of "standards of practice" by 
continuing education pro/lders selected tasks from the standards were used 
in both the design and evaluation of a six hour Introductory program on 
effective communication and pharmacy. Evaluation of program effect was 
based on a pre-test and a posttest. 

Hatch, J. A. (198J). Application of qualitative methods of program 

evaluation in education. Viewpoints in Teaching and Learning , 59(1), 1-11, 

This article explores the utilization of qualitative methodology in 
program evaluation, the rationale for using these approaches, and the 
potential qualitative applications within the evaluation. It also 
describes the "scientific- and "artistic" qualitative methodologies, 
suggests practical qualitative application, and calls for conceptual shift 
in methodology decision making in the evaluation. 

Health, R. W. & Brandon, P. R. (1982). An alternative approach to the 

evaluation of educational and social programs. Educational Evaluation a nd 
Policy Analysis , 4(4), 477-486. 

This paper discusses problems associated with the application of 
experimental methods to program evaluation, proposes an alternative 
approach (program attributes scale) and reports the procedures and results 
of the trial application of this alternative approach. 

Holt, Margaret E. & Courtenay, Bradley C. An Examination of Impact 
Evaluations. Continuum ; 49(1), 23-35. 

Presents guidelines for determining whether an Impact studies is 
desirable or feasible for evaluating continuing education programs. 
Describes evaluation methods and program aspects that can be measured. 

House, E. R. (1982). Alternative evaluation strategies in higher education. 
New Directions for Higher Education , Number 37. (Designing Academic 
Program Reviews); 10(1), 5-15. 

Different approaches to evaluation are described: system analysis, 
behavior objectives, professional review, and case study. In order for an 
evaluation system to operate, it must be perceived as being fair. The 
article makes the point that the legitimacy of the evaluation is dependent 
on the issue of fairness. 



Huffman, J. & Harris, J. (1981). Implications of the "input-outcome" 
research for the evaluation and accreditation of educational programs. 
North Central Association Quarterly , 56(1), 27-32. 

Reviews six studies that investigated the relationship between input 
variables (e.g. training, experience, education staff, suitability and 
availability of materials and teaching aids, and adequacy of facilities) 
and educational outcomes. Examines the implications of the study findings 
for the evaluation and accreditation of educational programs. 
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Jacobson, Harvey (1978) Framework for Evaluation, Indicators of Effort, 
Performance, Effects. New Directions for Institutional Advancement 
(Evaluating Advancement Programs) ; 1, 17-63, 

The problem of selecting appropriate Indicators for the evaluation of 
higher education programs Is discussed. A set of Indicators and a model 
for assessing institutional advancement programs are presented as a 
framework for the Institutional manager. 

Kemmls, S. (1982). Seven principles for program evaluation in curriculum 
development and innovation. Journal of Curriculum Studies , 14(3), 221-240. 

Proposes seven principles of program evaluation to serve as guidelines 
for designing evaluation procedures in Australian schools. The object of 
the evaluation is to generate useful information about the educational 
environment which can be used for decision making in the curriculum 
development process. 

Kennedy, M. M. (1983). The role of the In-house evaluator. Evaluation 
Review , 7(4), 519-541. 

In-house evaluators must develop an evaluation procedure that 
accommodate not only their own professional standards but also the 
problem- solving styles of the organizations that they serve. This article 
describes how a small group of evaluators have adapted their loiethods to 
their contexts. 

Knox, Alan B. (1979) Conclusions about Impact Evaluation. New Directions 
for Continuing Education ; 3, 117-21. 

There is sufficient evidence from Impact studies throughout the field 
to conclude that continuing education can have an Impact on evaluation. 
The challenges to Increase the proportion of programs that do so and to 
develop more efflcler*- procedures to assess impact. 

Lee, Barbara (19b5) Statistical Conclusion Validity and Ex-post Facto 

Designs, Practicality and Evaluation. Educational Evaluation and Policy 
Analysis ; 7(1), 35-45. 

Different evaluation models were applied to data from high school 
career education program to investigate problems in statistical conclusion 
validity and prograu effectiveness judgements. If potential threats to 
Internal validity are analyzed and protection strategies are developed more 
confidence in unplanned ex-post facto design using a naturalistic model is 
justified. 

Legare, Thomas L. (1980) Evaluating Educational Profram when Experimental 
Design is Inappropriate. Educational Research Quarterly ; 5(2), 37-49. 

Applying attribution theory principles to evaluation of educational 
programs. The affects of training for social service employes who are 
evaluated by trainees, training Instructors and agency supervisors. 
Findings confirmed a concept of attribution theory. Trainees and 
instructo>.s tended to have similar responses while agency supervisors 
responses were different. 
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Light, Richard J. (1980) Evaluation Methodology for Education Programs with 
Diffused Goals: Applications to Extension Evaluation. Education and Urban 
Society ; 13(1), 109-34, 

Discusses applications of evaluation methodology for educational 
programs to extension evaluation. Reviews alternative evaluation designs, 
makes specific suggestions for Interpreting past evaluation studies and 
present Ideas for future extension evaluation. 

Light, Richard J. (1984) Six Evaluation Issues that Synthesis Can Resolve 
Better than Single Studies. New Directions for Program Evaluation; 24, 



Using the examples from government program evaluation studies, six 
areas where research synthesis Is more effective than Individual studies 
are presented. Research synthesis can (1) help match treatment type with 
recipient type, (2) explain which treatment features matter, (3) explain 
conflicting results, (4) determine critical outcomes, (5) assess treatment 
stability and (6) assess research design. 

Llndvall, C. M. & Nltko, A. J. (1981), Basic considerations In assessing the 
validity of evaluation designs. Educational Evaluation and Policy 
Analysis , 3(4), 49-60, 

Designs for evaluation studies of educational programs should provide 
valid and defensible Inferences. Design problems may be resolved by 
creatively utilizing features of specific evaluations in designing unique 
conditions that permit valid Inferences. 

Martin, Ann M. (1979) Program Evaluation, Some Federal Imperatives. Journal 
of Studies and Technical Careers ; 1(2), 143-51, 

A major problem facing program evaluation of vocational education Is a 
lack of a national manpower policy. This concern involves workers 
acquiring and developing occupational skills which are needed by society 
and demanded by employers. Federal legislation specifications spell out 
program requirements which educator's muvt keep In mind. 

Mathls, William. (1980) Evaluating the Policy Implications. CEDR-Quarterly ; 
13(2), 3-6, 22, 

A number of ways in which evaluations may be biased are Illustrated (1) 
policy and political purposes for evaluations; (2) sources of policies or 
political bias prior to evaluation activities, (3) limitations and biases 
within the evaluation itself, (4) utilization of evaluation information In 
the policy process. 

Matkln, R, E. (1982). Program evaluation strategies for private-f or-prof it 
rehabilitation. Rehabilitation Counseling Bulletin , 25(5), 268--277, 

Advocates the need for a systematic approach to evaluating services for 
those participating in private rehaMiitaticn. Uses examples of services, 
that are performed in the privates sector. Discusres summative and 
foraiatlve evaluctioa strategies. 



57-73, 
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Mullen, ?• D» & Iverson, D* (1982) • Qualitative methods for evaluative 
research in health education programs. Health Education , 13(3), 11-18. 

Qualitative methods can play a significant role in both formative and 
summative evaluations. There are at least six functions of the qualitative 
methods in education* Tables and figures present information and 
characteristics of the qualitative and quantitative methods, selected 
qualitative evaluation models and a check list of evaluation situations for 
which qualitative methods are appropriate. 

Munitz, Barry A* & Wright, Douglas J. (1980) Institutional Approaches to 
Academic Program Evaluation. New Directions for Institutional Research , 27 
(Academic Program Evaluation) , 7(3) , 21-42. 

The imperatives of institutional planning in an environment of scarce 
resources have brought program evaluation activities under more systematic 
processes serving varied needs. Evaluation processes at Michigan State 
University, the University of Michigan and the University of Houston are 
examined. 

Munro, B. H. (1983). A useful model for program evaluation. Journal of 
Nursing Administration , 13(3), 23-26. 

This model for program evaluation, that WcS originally designed for use 
by educational evaluators has been found useful for the evaluation of 
health care delivery programs. The model provides the basis for 
comprehensive evaluation and could serve as a useful tool for nurse 
administrators in the ongoing evaluation oi the programs they supervise. 

Newman, I. & Deitchman, R. (1983). Evaluation-research: A suggested move 
toward scientific and community credibility. Journal of Studies in 
Technical Careers , 5(4), 289-298. 

Discusses the relationship between research and evaluation. Presents 
several evaluation models currently used and discusses considerations for 
each of them. 

Ross, John A. (1980) Decision Ruler and Program Evaluation. Evaluation 
Review ; 4(1), 59-74. 

An example is provided of an evaluation in which the formulation of 
explicit decision rules was a central activity. The case for and against 
the use of decision rules in program evaluation is considered and the 
appropriate context identified. Implications for the rights of decision 
makers are addressed. 

Rossi, P. H. 6f Berk, R. A. (1981). An overview of evaluation strategies and 
procedures. Human Organization , 40,(4), 287-299, 

Provides a detailed introduction and the variety of purposes for which 
evaluation research may be used into the wide range of methods currently 
explored in the field. Uses specific examples to provide concrete 
illustrations iii bo*:h goals of the evaluation research and the methods used. 
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Rossl, Robert A, & McLalghlin, Donald H. (1979) Establishing Evaluation 
Objectives, The Evaluation Quarterly ; 3(3), 331-46* 

Establishing evaluation objectives requires formal, frequently and 
extensive interaction among program evaluator's and administrators to 
obtain from administrators the information necessary to make the most 
effective choices in planning and carrying out evaluations. Three 
techniques are described that can be used to insure that this interaction 
is constructive* 

Saracho, 0. (1982), New dimensions in evaluating the worth of a program. 
Evaluation , 103(1), 74-78. 

Evaluation is essential to assess the worth of a program and to provide 
suggestions for modif ications. An ideal evaluation program can be designed 
by implementing the expert's best ideas, provided they meet the needs of 
the program and by involving those who are directly and Indirectly involved 
In the program, 

Scott, Robertt A. (1980-81) Quality: A Program's Reviews Kissing Link. 
College Board Review ; 118, 18-21, 30. 

How to define quality and how to improve and clarify present standards 
of judgement and assessment in education are discussed. Three forms of 
external program review are identified; regional accreditation, 
professional society accreditation and the state agency evaluation. 
Quality is seen as representing the level of attainment over appropriate 
goals. 

Simpson, William A. (1985) Easing the Pain of Program Review: Departments 
take the Initiative. Educational Record ; 66(2), 40-42, 

Renewed interests in program reviews has developed in recent years as 
resources for higher education have declined and demand for accountability 
has increased. University level program review and department and unit 
level program review are discussed. 

Slimmer, V. M. (1983), Evaluating Programs in Higher Education. Assessment 
and Evaluation in Higher Education , 8(1), 52-65. 

A model for the evaluation of an ongoing college program is presented 
that integrates six instrument assessing constituent groups satisfactions 
or dissatisfactions: current student satisfactions survey, graduate survey, 
administrator's perception survey, questionnaire on the program's strengths 
and weaknesses and visibility within the state. 

Slotnick, H. B. (1982). A simple technique for collecting, analyzing and 
interpreting evaluation data. Evaluation and Health Profession s, 5(3), 
245-58, 

Describes the best liked/least liked technique for collecting 
evaluation information. The technique is composed of stages (data 
collection, data analysis and data interpretation) that are characterized 
by their specific use. The techniques used can be narrowly or broadly 
focused, and it can be used for achievem»-nt testing or for affective 
measurement. 
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Smartt, Steven, (1984) Linking Program Reviews to the Budget. New Direction 
for Higher Education ; 48, (Financial Incentives for Academic Quality 
Person); 12(4), 43-56, 

Three types of state level roles In higher education program review as 
It relates to budgeting are outlined. Ideas from the literature about the 
efficacy of program budgeting are summarized, and North Carolina's use of 
state level reviews for program Improvement and budgeting is described. 

Smith, Donald K. (1980) Multi-campus System Approaches to Academic Program 
Evaluation. New Directions for Institutional Research , 27 (Academic 
Program Evaluation); 7(3), 43-67. 

Multi-campus systems should proceed carefully in the process of 
academic program review with an eye toward asking and answering q tstioas 
mostly likely to cause institutions to confront the more fundamental 
questions of purpose and coherence 

Smith, N. L. (1981). Evaluation design as preserving valid qualities in 
evaluation studies. Studies in Educational Evaluation , 7(3), 229-37. 

The evaluatioc design is seen as an Important factor in determining the 
quality of an evaluation; values and valued qualities permeate the 
evaluation design. Mechanisms for evaluations (in process reviews > 
follow-up studies, etCv) are dealt with and familiarity with financial, 
managerial and ethical procedures are stressed in the eveluation design. 

Smith, Nick L. & Caulley, Darrel N. (1979) Post-tvaluatloxi Determination of 
a Program's Generalized Ability*. Evaluation of Program Planning; an 
I ntemational Journal ; 2(4), 297-02. 

Literature on the generalizability of program effects focusses on the 
priori development of evaluation designs which enables certain 
generalizations to be made. Secondary analysis procedures which can be 
employed using existing evaluation data to estimate a program* s 
generalizability when follow up studies are not feasible are suggested. 

Tallmadge, G. Kasten. (1979) Avoiding Problems in Evaluation. Journal of 
Career Education ; 5(4), 300-08. 

DiscuGsas the need for awareness of real and potential problem^i in 
educational evaluation approaches. Kinds of problems there are with true 
experimental designs and with quasi experimental designs and the criteria 
applied by the Joint Dissemination Review Tauel to which review of 
educational programs. 

Tyler, Ralph W. (1980) A brief overview of progriim evaluation. Journal of 
Cooperative Education ; 16(3), 7-15. 

Critical issues in the process of program evaluation are (1) 
identification and definition of objectives ani! outcomes, (2) development 
of valid and reliable forms of assessment, (3) development of methods, data 
analysis and interpretation, and (4) extension of the use of evaluation 
results. 
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VoeltZ) Luann A« et al* (1979) A Needs Assessment of Special Education 
In-service Training Priorities for Teachers, Educational Perspectives ; 



The University of Hawaii, and the State Department of Education have 
already Identified a service delivery model called Ho'okoho to meet the 
personnel training requirements of PL 94-142* This article reports results 
of the program's first survey to assess in-service training priorities of 
both regular and special educators* 

Wilcox, B* (1982), School self-evaluation: The benefits of a more focused 
approach. Educational Review , 34(3), 185-93* 

This paper outlines some of the general characteristics of current 
public school self-evaluation schemes and suggests that chey may be useful 
from an organizational standpoint. The construction and use of a 
self-devaluation instrument which was formed by using this approach is 
briefly described. 

Windoe, Charles* (1979) Developmental Trends in Program Evaluation* 

Evaluation and Px^ogram Planning: an International Journal ; 2(3) ^ 193-96* 

It suggested that the developmental phases of program evaluators 
replicate those of the profession of program evaluation* A trend in 
program evaluation from progt^sia amelioration to progrosi advocacy and 
ultimately to program accountability is projected. 

Wolf, Robert L* (1979) Tht Use of Judicial Evaluation Methods in the 
Fonaui^ition of Educational Policy* Educ^tioiial Evaluation and Policy 
Analysis ; 2(3), 19-28* 

The. judicial evaluation model JEM, is useful for aiding decision-making 
bodies to formulate policies* Broad public participation in complax issues 
is provided by JEM. The conceptualization policy, formulation stages and 
participant roles are identified and the lacdel is applied to two case 
studies. 

Young, Kenneth E. L Chambers, Charles Mc (1980) Accrediting Agency 
Approaches to Academic Program Svaiuation* Nev Directions for 
Institutional Research , 27 (Academic Program Evaluation) ? 7(3). 89-103. 

Nongovermental accreditation of institutions and pro^^rams are uniquely 
American process is discussed* Institutions are shown to bear a large 
responsibility for making accreditation work as a major tool for 
improvement. Deaccreditation process specialized accreditation and future 
developments are described. 
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